Layer based hrd buffer management for scalable hevc

ABSTRACT

The buffer management methods simplify the complexity of STD buffer management for HEVC and make is easy to implement HEVC in deployed AVC/MPEG-2 networks (as legacy re-multiplexers are able to be used for re-purposing HEVC content). The buffers for base and enhancement layers are also also be managed independently before re-assembly and this simplifies the STD model. Re-assembly is still implemented before decoding an enhanced HEVC video stream.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority under 35 U.S.C. § 119(e) of the U.S.Provisional Patent Application Ser. No. 61/846,552, filed Jul. 15, 2013and titled, “LAYER BASED HRD BUFFER MANAGEMENT FOR SCALABLE HEVC” andU.S. Provisional Patent Application Ser. No. 61/889,980, filed Oct. 11,2013 and titled, “MULTILAYER HRD MANAGEMENT,” which are both herebyincorporated by reference in their entireties for all purposes.

FIELD OF THE INVENTION

The present invention relates to the field of imaging. Morespecifically, the present invention relates to video coding.

BACKGROUND OF THE INVENTION

The buffer management in Advanced Video Coding (AVC)-based scalability(Scalable Video Coding (SVC) and Multiview Video Coding (MVC)) requiredan extension to system STD buffer model and introduced an additionallayer of complexity to re-purposing and re-distribution equipment. Theextensions required management of both the base layer buffer and bufferwith base and enhancement layer/layers at the same time in bothtransmission and decoding equipment.

SUMMARY OF THE INVENTION

The buffer management methods simplify the complexity of STD buffermanagement for HEVC and make is easy to implement HEVC in deployedAVC/MPEG-2 networks (as legacy re-multiplexers are able to be used forre-purposing HEVC content). The buffers for base and enhancement layersare also also be managed independently before re-assembly and thissimplifies the STD model. Re-assembly is still implemented beforedecoding an enhanced HEVC video stream.

In one aspect, a method programmed in a non-transitory memory of adevice comprises acquiring a video and transmitting a bitstream of thevideo including signaling a maximum bit rate, average bit rate andmaximum coded picture buffer size allocated for each layer in thebitstream. The method further comprises implementing an extension andparameters that are configured to be defined as whole-sequence level orsub-stream level. The method further comprises managing a system bufferfor each layer independently without having to re-assemble lower layers.The method further comprises managing buffers for base and enhancementlayers of the video before re-assembly. The method further comprisesimplementing a multilayer decoder. The method further comprisesassociating buffering period and picture timing supplemental enhancementinformation messages with each layer of the video. The method furthercomprises utilizing a same picture timing supplemental enhancementinformation message for all layers of the video for synchronization ofremoval times within a layer set, and utilizing different bufferingperiod supplemental enhancement information messages for each layer ofthe video. The device is selected from the group consisting of apersonal computer, a laptop computer, a computer workstation, a server,a mainframe computer, a handheld computer, a personal digital assistant,a cellular/mobile telephone, a smart phone, a smart appliance, a gamingconsole, a digital camera, a digital camcorder, a camera phone, anportable music player, a tablet computer, a video player, a DVDwriter/player, a high definition video writer/player, a television and ahome entertainment system.

In another aspect, a system programmed in a non-transitory memory of acamera device comprises a sensor configured for acquiring a video and atransmitting component configured for transmitting a bitstream of thevideo including signaling a maximum bit rate, average bit rate andmaximum coded picture buffer size allocated for each layer in thebitstream. The transmitting component is further configured forimplementing an extension and parameters that are configured to bedefined as whole-sequence level or sub-stream level. The transmittingcomponent is further configured for managing a system buffer for eachlayer independently without having to re-assemble lower layers. Thetransmitting component is further configured for managing buffers forbase and enhancement layers of the video before re-assembly. Thetransmitting component is further configured for implementing amultilayer decoder. The transmitting component is further configured forassociating buffering period and picture timing supplemental enhancementinformation messages with each layer of the video. The transmittingcomponent is further configured for utilizing a same picture timingsupplemental enhancement information message for all layers of the videofor synchronization of removal times within a layer set, and utilizingdifferent buffering period supplemental enhancement information messagesfor each layer of the video.

In another aspect, an apparatus comprises a sensor configured foracquiring a video, a non-transitory memory for storing an application,the application for: transmitting a bitstream of the video includingsignaling a maximum bit rate, average bit rate and maximum coded picturebuffer size allocated for each layer in the bitstream and a processingcomponent coupled to the memory, the processing component configured forprocessing the application. The application is further configured forimplementing an extension and parameters that are configured to bedefined as whole-sequence level or sub-stream level. The application isfurther configured for managing a system buffer for each layerindependently without having to re-assemble lower layers. Theapplication is further configured for managing buffers for base andenhancement layers of the video before re-assembly. The application isfurther configured for implementing a multilayer decoder. Theapplication is further configured for associating buffering period andpicture timing supplemental enhancement information messages with eachlayer of the video.

The application is further configured for utilizing a same picturetiming supplemental enhancement information message for all layers ofthe video for synchronization of removal times within a layer set, andutilizing different buffering period supplemental enhancementinformation messages for each layer of the video.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a diagram of AVC and SVC decoding according to someembodiments.

FIG. 2 illustrates an example of the system buffer model for HEVClayered coding and this is the same as that specified for SVC and MVC.

FIG. 3 illustrates a diagram of scenarios of decoders forSHVC/multi-view according to some embodiments.

FIG. 4 illustrates a diagram of a 2-decoder model case according to someembodiments.

FIG. 5 illustrates an example of a multilayer HRD buffer flow model froma system STD operation perspective according to some embodiments.

FIG. 6 shows a generalization of HRD buffer flow according to someembodiments.

FIG. 7 illustrates a structure of byte streams and NAL unit streams forconformance checks according to some embodiments.

FIG. 8 illustrates an HRD buffer model according to some embodiments.

FIG. 9 illustrates a diagram of a bitstream-partition-specific CPBoperation according to some embodiments.

FIG. 10 illustrates a flowchart of implementing buffer managementmethods according to some embodiments.

FIG. 11 illustrates a block diagram of an exemplary computing deviceconfigured to implement the buffer management methods according to someembodiments.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Described herein are options to reduce the buffer model complexity inHigh Efficiency Video Coding (HEVC)-based scalability.

The scalability model as introduced in SVC (and also used in MVC) usesAVC base layer and adds scalable enhancements to achieve temporal,Signal-to-Noise Ratio (SNR) and spatial improvements. Typical SVCapplications targeted use cases where bit rate savings were achievedover simulcast of ‘base layer and enhancement without scalability.’ MVCprovided view enhancements using a similar model.

Some of the application use cases required transmission of AVC baselayer stream to base layer decoder while the enhancement layer wastransmitted separately where the SVC/MVC decoder assembled the base andenhancement layers before decoding the enhanced data.

FIG. 1 illustrates a diagram of AVC and SVC decoding according to someembodiments. The base layer video stream used a different profile andlevel than the [base+enhancement layer] (usually the enhancement layerbeing higher level than base layer), and each used a different CodedPicture Buffer (CPB) size and bit rate for Hypothetical ReferenceDecoder (HRD) buffer management. The video standard (and in turnsystems) did not specify an independent buffer size and rate for eachenhancement layer and the specification defined a scheme where the baselayer and [base+enhancement layer] buffers were managed in parallel tomake sure they did not overflow or underflow. This management wasrequired at both the encoding end as well as decoders. The HRD model forSVC and MVC did not address such parallel CPB management of base layerand [base+enhancement layer]. The HRD model defined how to extract baselayer video from the full SVC/MVC access unit and then manage the baselayer.

The transport part of SVC/MVC had to support the STD managementindependently for base and [base+enhancement layer] as STD does not havethe ability to extract base layer data from full SVC access units andthis capability would have required the base layer decoders to supportthe larger buffer sizes that included the base and enhancement layerdata. The STD extension in systems requires a virtual re-assembly bufferand the management of multiple buffers after re-assembly. This has madethe STD more complex for SVC and MVC transport and makes re-multiplexingvery difficult, complex and not implementable with legacyre-multiplexers. The defined compression of enhancement layer requiredvariable bit rate and buffer allocation between base layer andenhancement layer to achieve best results. For example, if it wasrequired to use a higher buffer size and rates for base layer, theequivalent was adjusted in enhancement layer and vice versa. However,the buffer size and rates for base layer or [base+enhancement] layercould not exceed the maximum limits for the level specified.

FIG. 2 illustrates an example of the system buffer model for HEVClayered coding and this is the same as that specified for SVC and MVC.

Simplify Buffer Management Issues in HEVC Scalability

HEVC scalability includes additional use cases than just ‘simulcast’ andis more efficient in compression of enhancement layers. In all of thesecases, it will be difficult to manage base layer and [base+enhancementlayer] buffers in parallel to achieve conforming system buffer behaviorif HEVC scalability adopts a similar HRD buffer model as AVCscalability. The schemes described herein simplify system buffermanagement where the parallel buffer management is able to be avoided byuse of a new SEI and an extension to HRD parameters.

HEVC_scalability SEI: The proposed SEI (based on SVC scalability infoSEI or MVC view_scalability SEI) signals the maximum bit rate, averagebit rate and maximum CPB buffer size allocated for each of the layers inthe video stream. This is also be mapped into the system layer using adescriptor and allows implementation of independent single buffer STDmodel in systems as well as HRD for base and enhancement layers.

HRD extensions: Currently, HEVC HRD parameters are able to be definedeither as whole-sequence level or sub-stream level. There might be thecases where system buffer is able to be managed for each layer/viewindependently without having to re-assemble the lower layer/views, butthe current HRD specification does not support such scenario. Describedherein uses the scalable HEVC extension in vps_extension( ) andsps_extension( ) to convey HRD parameters for a single-layer model ormulti-layer model (like SVC or MVC) or both.

The methods simplify the complexity of STD buffer management for HEVCand make it easy to implement HEVC in deployed AVC/MPEG-2 networks (aslegacy re-multiplexers are able to be used for re-purposing HEVCcontent). The buffers for base and enhancement layers are also able tobe managed independently before re-assembly, and this simplifies the STDmodel. Re-assembly is still implemented before decoding an enhanced HEVCvideo stream.

HEVC Scalability SEI

A new suffix SEI to HEVC to signal layer specific is utilizedinformation such as maximum bit rate, average bit rate and maximum CPBbuffer size. The following table provides an example of the SEI messagesyntax where the parameters are able to be signaled for a single layer

(vps_max_layers_minus1=0) or multiple layers.

Table 1 shows HEVC scalability information SEI message syntax.

Descriptor HEVC_scalability_info (payloadSize) { active_vps_id u(4) for(i=0; i<= vps_num_layer_sets_minus1; i++) { for(j=1;j<=vps_max_layer_id; j++) { if(layer_id_included_flag[i][j]) {layer_level_info_present_flag[i][j] u(1)if(layer_level_present_flag[i][j]) layer_level_idc[i][j ] u(8)layer_hrd_info_present_flag[i][j] u(1)layer_sub_pic_hrd_info_present_flag[i][j ] u(1)if(layer_hrd_info_present_flag[i][j]{ layer_bit_rate_value_minus1[i][j]ue(v) layer_cpb_size_value_minus1[i][j] ue(v) }if(layer_sub_pic_hrd_info_present_flag[i][j]) {layer_bit_rate_du_value_minus1[i][j] ue(v)layer_cpb_size_value_minus1[i][j] ue(v) } } } }active_vps_id: identifies an active VPS that contains the layerinformation. The value of active_vps_id shall be equal to the value ofvideo_parameter set id of the active VPS for the VCL NAL units of theaccess unit containing the SEI message.layer_level_info_present_flag[i][j]: specifies whether the levelinformation about layer index j in layer set i is present.layer_level_idc[i][j]: indicates a level to which jth layer in layer seti in the CVS conforms as specified.layer_hrd_info_present_flag[i][j]: specifies whether the hrd informationabout layer indexj in layer set i is present.layer_hrd_sub_picture_info_present_flag[i][j]: specifies whether thesub-picture hrd information about layer index j in layer set i ispresent.layer_bit_rate_value_minus1[i][j]: (together with cpb_size_scale)specifies the maximum input bit rate of layer index j in layer set i.layer_cpb_size_du_value_minus1[i][j]: (together with layer_cpbsize_scale) specifies the CPB size of layer index j in layer set I CPBwhen CPB operates at sub-picture level.

Table 2 shows HEVC scalability information SEI message syntax.

HEVC_scalability_info (payloadSize) { Descriptor  active_vps_id u(4) layer_bit_rate_scale u(4)  layer_cpb_size_scale u(4)  for (i=0; i<=vps_num_layer_sets_minus1; i++) {   for(j=1; j<=vps_max_layer_id; j++) {   if(layer_id_included_flag[i][j]) {    layer_level_info_present_flag[i][j] u(1)   if(layer_level_present_flag[i][j])     layer_level_idc[i][j] u(8)   layer_hrd_info_present_flag[i][j] u(1)   if(layer_hrd_info_present_flag[i][j]{    layer_bit_rate_value_minus1[i][j] ue(v)    layer_cpb_size_value_minus1[i][j] ue(v)    layer_ave_bit_rate_value_minus1[i][j] ue(v)    layer_max_cpb_size_value_minus1[i][j] ue(v)    layer_bitrate_calc_window[i][j] ue(v)    }   }  } }active_vps_id: identifies an active VPS that contains the layerinformation. The value of active_vps_id shall be equal to the value ofvideo_parameter set id of the active VPS for the VCL NAL units of theaccess unit containing the SEI message.layer_bit_rate_scale: has similar semantics to bit rate scalelayer_cpb_size_scale: has similar semantics to cpb_size_scalelayer_level_info_present_flag[i][j]: specifies whether the levelinformation about layer index j in layer set i is present.layer_level_idc[i][j]: indicates a level to which jth layer in layer seti in the CVS conforms as specified.layer_hrd_info_present_flag[i][j]: specifies whether the hrd informationabout layer indexj in layer set i is present.layer_bit_rate_value_minus1[i][j]: (together with layer_bit_rate_scale)specifies the maximum input bit rate of layer index j in layer set i.Has similar semantics as bit_rate_value minus1.layer_avg_bit_rate_value_minus1[i][j]: (together withlayer_bit_rate_scale) specifies average value of layer index j in layerset i.layer_max_cpb size_value_minus1[i][j]: (together with layer_cpbsize_scale) specifies maximum cpb size value of layer index j in layerset i.max_bitrate_calc_window[i][j]: specifies the window to calculate maximumbit rate of layer index j in layer set i.Table 3 shows HEVC scalability information SEI message syntax.

HEVC_scalability_info (payloadSize) { Descriptor  active_vps_id u(4) layer_bit_rate_scale u(4)  layer_cpb_size_scale u(4)  for (i=0; i<=vps_max_layer id; i++) {   if(layer_id_included_flag[i]) {   layer_level_info_present_flag[i] u(1)  if(layer_level_present_flag[i])    layer_level_idc[i] u(8)  layer_hrd_info_present_flag[i] u(1)  if(layer_hrd_info_present_flag[i]{    layer_bit_rate_value_minus1[i]ue(v)    layer_cpb_size_value_minus1[i] ue(v)   layer_ave_bit_rate_value_minus1[i] ue(v)   layer_max_cpb_size_value_minus1[i] ue(v)   layer_bitrate_calc_window[i] ue(v)    }   }  } }active_vps_id: identifies an active VPS that contains the layerinformation. The value of active_vps_id shall be equal to the value ofvideo_parameter set id of the active VPS for the VCL NAL units of theaccess unit containing the SET message.layer_bit_rate_scale: has similar semantics to bit rate scalelayer_cpb_size_scale: has similar semantics to cpb_size_scalelayer_level_info_present_flag[i]: specifies whether the levelinformation about layer index i.layer_level_idc[i]: indicates a level to which ith layer in the CVSconforms as specified in.layer_hrd_info_present_flag[i]: specifies whether the hrd informationabout layer index i is present.layer_bit_rate_value_minus1[i]: (together with layer_bit_rate_scale)specifies the maximum input bit rate of layer index i. Has similarsemantics as bit_rate_value_minus1.layer_avg_bit_rate_value_minus1[i]: (together with layer_bit_rate_scale)specifies average value of layer index.layer_max_cpb_size_value_minus1[i]: (together with layer_cpb_size scale)specifies maximum cpb size value of layer index i.max_bitrate_calc_window[i]: specifies the window to calculate maximumbit rate for layer index i.

HRD Extensions

In FIG. 3, the bitstream contains 2 layers or views. It is possible forthe whole bitstream to be decoded with one decoder (scenario 1) or eachlayer is able to be decoded with separate decoders (scenario 2). FIG. 4illustrates a diagram with a 2-decoder model case, where each layer isdecoded.

hrd_parameters( ) are defined differently with scenario 1 and 2: inscenario 1, parameters of both BL and EL should be included within onehrd_parameters( ) while in scenario 2 parameters of BL and EL should beincluded in separate hrd_parameters. However, in the currenthrd_parameters( ) syntax, specification is allowed to be specified foreach layer set, which means that only scenario 1 is supported.

To solve this problem, the syntax is modified so that scenario-2-typehrd parameters( ) are able to be optionally transmitted invps_extension( ) and sps_extension( ).

Table 4 shows vps_extension( ) syntax.

vps_extension( ) { Descriptor  . . .  for(i=0;i<=vps_num_layer_sets_minus1; i++)   for(j=1; j<=vps_max_layer_id; j++)   if(layer_id_included_flag[i][j]) {    layer_set_hrd_layer_info_present_flag[i][j] u(1)   if(layer_set_hrd_layer_info_present_flag[i][j]) {    hrd_paramaters(cprm_present_flag[i], vps_max_sub_layers_minus1)   } . . . }layer_set_hrd_layer_info_present_flag[i][j]: specifies whether the hrdinformation about layer index j in layer set i is present.Table 5 shows vps_extension( ) syntax.

vps_extension( ) { Descriptor  . . .  for(i=0; i<=vps_num_hrd_parameters− 1; i++)   1stIdx = hrd_layer_set_idx[i]   for(j=1;j<=vps_max_layer_id; j++)    if(layer_id_included_flag[1stIdx][j]) {    layer_set_hrd_layer_params_present_flag[i][j] u(1)   if(layer_set_hrd_layer_params_present_flag[i][j]) {    hrd_paramaters(cprm_present_flag[i], vps_max_sub_layers_minus1)   } }  . . . }Table 6 shows vps_extension( ) syntax.

vps_extension( ) { Descriptor  . . .  for(i=0; i<=vps_num_hrd_parameters− 1; i++)   1stIdx = hrd_layer_set_idx[i]   for(j=1;j<numLayersInIdList[1stIdx]; j++)    if(output_layer_flag[1stIdx][j]) {    layer_set_hrd_layer_params_present_flag[i][j] u(1)   if(layer_set_hrd_layer_params_present_flag[i][j]) {    hrd_paramaters(cprm_present_flag[i], vps_max_sub_layers_minus1)   } }  . . . }Table 7 shows sps_extension( ) syntax.

sps_extension( ) { Descriptor  . . .   layer_set_hrd_layer_info_present_flag u(1)   if(layer_set_hrd_layer_info_present_flag) {     hrd_parameters(1,sps_max_sub_layers_minus1)   }  . . . }layer_set_hrd_layer_info_present_flag equals to 1 specifies thathrd_parameters( ) that is applied just to the current layer which is apart of layer set is additionally contained in the CVS, and equals to 0specifies that hrd_parameters( ) that is applied just to the currentlayer which is a part of layer set is not contained.

The methods described herein are for transmission of individual bufferparameters for HEVC scalable layers so that the system buffer managementis able to be simplified. The first scheme includes a new SEI totransmit the information while the second scheme includes extensions toHRD parameters in vps_extension( ) and sps_extension( ).

Multilayer HRD Management

Layer-based HRD buffer flow management for a multilayer operation isbeing described herein. Multi-layer representation of a bitstream is aninherent characteristic of scalability models as practiced, in SHVC andMV-HEVC, to allow temporal, spatial and view enhancements. In addition,a hybrid scalable model is also introduced allowing the transmission ofAVC base layer stream to an AVC base layer decoder while an HEVCenhancement layer is transmitted to an HEVC decoder, separately.Traditional HRD buffer management approaches identify HRD parameters andnumber of HRD models on the whole e.g., for combined layers as opposedto on a per layer basis.

FIG. 5 shows an example of a multilayer HRD buffer flow model from asystem STD operation perspective. As shown, it includes a PID-baseddemux to map a multi-layered transport stream into its constituentlayers (2 layers in this example). Each layer is then fed into TB & MBbuffers to generate elementary streams as inputs to the CPB buffers.FIG. 6 shows a generalization of HRD buffer flow.

Table 8 shows vps_extension( ) syntax.

vps_extension( ) { Descriptor  . . .    sub_cpb_parameters_table( )  . ..  } sub_cpb_parameters_table( ) . . .   for(i=0;i<=vps_num_layer_sets_minus1; i++)    sub_cpb_info_present_flag[i] u(1)   if(sub_cpb_info_pressent_flag[i] {     for(k=0;k<vps_num_hrd_parameters && i!=hrd_layer_set_idx[k]; k++)      for(j=1;j<=vps_max_layer_id; j++) {       if(layer_id_included_flag[i][j] {     hrd_paramaters(cprm_present_flag[k], vps_max_sub_layers_minus1)   }   }  } }  . . . }sub_cpb_info_present_flag[i]: sub_cpb_info_present_flag[i]=1 indicatesCPB parameters for each layer in the layer set i are given.sub_cpb_info_present_flag[i]=0 indicates CPB parameters for each layerare not present.

For BP & PT SEI messages, because of the strict output timingrequirements, there is a single removal time and accordingly a single PTSEI message to be associated with all the layers. BP SEI messages areable to be associated with each layer for sub_CPB buffer management.

Table 9 shows a modified SEI message syntax.

multilayer_hrd_info (payloadSize) { Descriptor  active_vps_id u(4)  for(i=1; i<= vps_num_layer_id; i++) {   layer_hrd_info_present_flag[i] u(1)  if(layer_hrd_info_present_flag[i]{    layer_max_bitrate_value[i] u(16)   layer_max_cpb_size_value[i] u(16)   }  } }

Described herein is the HRD and its use to check bitstream and decoderconformance. Two types of bitstreams or bitstream subsets are subject toHRD conformance checking. The first type, called a Type I bitstream, isa NAL unit stream containing only the VCL NAL units and NAL units withnal_unit_type equal to FD_NUT (filler data NAL units) for all accessunits in the bitstream. The second type, called a Type II bitstream,contains, in addition to the VCL NAL units and filler data NAL units forall access units in the bitstream, at least one of the following:

additional non-VCL NAL units other than filler data NAL units,

all leading_zero_8bits, zero_byte, start_code, start_code_prefix_one_3bytes, and trailing_zero_8bits syntax elements that form a byte streamfrom the NAL unit stream. FIG. 7 shows the types of bitstreamconformance points checked by the HRD.

The syntax elements of non-VCL NAL units (or their default values forsome of the syntax elements), used for the HRD, are specified.

Two types of HRD parameter sets (NAL HRD parameters and VCL HRDparameters) are used. The HRD parameter sets are signaled through thehrd_parameters( ) syntax structure, which may be part of the SPS syntaxstructure or the VPS syntax structure.

Multiple tests are used for checking the conformance of a bitstream,which is referred to as the bitstream under test. For each test, thefollowing steps are applied in the order listed:

1. An operation point under test, denoted as TargetOp, is selected. Thelayer identifier list OpLayerIdList of TargetOp includes the list ofnuh_layer_id values, in increasing order of nub_layer_id values, presentin the bitstream subset associated with TargetOp, which is a subset ofthe nuh_layer_id values present in the bitstream under test. Thevariable TargetLayerSetIdx is set equal to index of the layer setspecified in the active VPS that exactly includes the layers inOpLayerIdList. The OpTid of TargetOp is equal to the highest TemporalIDpresent in the bitstream subset associated with TargetOp.

2. TargetDecLayerIdList is set equal to OpLayerIdList of TargetOp,HighestTid is set equal to OpTid of TargetOp, and the sub-bitstreamextraction process as specified is invoked with the bitstream undertest, HighestTid, and TargetDecLayerIdList as inputs, and the output isassigned to BitstreamToDecode.

3. When at least one bitstream partition HRD SEI message is present,bitstream-specific CPB operation or bitstream-partition-specific CPBoperation is selected and both CPB operations shall be tested forchecking the conformance of a bitstream. When bitstream-specific CPBoperation is tested, the subsequent steps apply for the bitstream undertest. When bitstream-partition-specific CPB operation is tested, thesubsequent steps apply to each bitstream partition of the bitstreamunder test, referred to as the bitstream partition under test. Whenbitstream-partition-specific CPB operation is tested and the input tothe HRD is a bitstream, the bitstream partitions are derived with thedemultiplexing process for deriving a bitstream partition.

4. The hrd_parameters( ) syntax structure and thesub_layer_hrd_parameters( ) syntax structure applicable to TargetOp areselected as follows:

If bitstream-specific CPB operation is tested, the following applies:

-   -   If TargetDecLayerIdList contains all nuh_layer_id values present        in the bitstream under test, the hrd_parameters( ) syntax        structure in the active SPS (or provided through an external        means) is selected.    -   Otherwise, the hrd_parameters( ) syntax structure in the active        VPS (or provided through some external means) that applies to        TargetOp is selected.    -   Otherwise, the hrd_parameters( ) syntax structure that applies        to the bitstream partition under test in the bitstream partition        HRD SEI message that is included in a scalable nesting SEI        message that applies to TargetOP (or provided through some        external means) is selected.

Within the selected hrd_parameters( ) syntax structure, ifBitstreamToDecode is a Type I bitstream, thesub_layer_hrd_parameters(HighestTid) syntax structure that immediatelyfollows the condition “if (vcl_hrd_parameters_present_flag) is selectedand the variable NalHrdModeFlag is set equal to 0; otherwise(BitstreamToDecode is a Type II bitstream), thesub_layer_hrd_parameters(HighestTid) syntax structure that immediatelyfollows either the condition “if(vcl_hrd_parameters_present_flag)” (inthis case the variable NalHrdModeFlag is set equal to 0) or thecondition “if(nal_hrd_parameters_present_flag)” (in this case thevariable NalHrdModeFlag is set equal to 1) is selected. WhenBitstreamToDecode is a Type II bitstream and NalHrdModeFlag is equal to0, all non-VCL NAL units except filler data NAL units, and allleading_zero_8bits, zero_byte, start_code_prefix_one_3 bytes, andtrailing_zero_8bits syntax elements that form a byte stream from the NALunit stream, when present, are discarded from BitstreamToDecode, and theremaining bitstream is assigned to BitstreamToDecode.

5. An access unit associated with a buffering period SEI message(present in BitstreamToDecode or available through external means)applicable to TargetOp is selected as the HRD initialization point andreferred to as access unit 0. An applicable buffering period SEI messageis selected from access unit 0 as follows:

If bitstream specific CPB operation is tested, the following applies:

-   -   If TargetDecLayerIdList contains all nuh_layer_id values present        in the bitstream under test, a non-nested buffering period SEI        message is selected.    -   Otherwise, a buffering period SEI message included in the        scalable nesting SEI message with bitstream_subset_flag equal to        1 and applicable to TargetOp is selected.    -   Otherwise, a buffering period SEI message included in the        bitstream partition nesting SEI message applicable to the        bitstream partition under test is selected.

6. For each access unit in BitstreamToDecode starting from access unit0, a buffering period SEI message (present in BitstreamToDecode oravailable through external means) that is associated with the accessunit and applies to TargetOp is selected, a picture timing SEI message(present in BitstreamToDecode or available through external means notspecified) that is associated with the access unit and applies toTargetOp is selected, and when SubPicHrdFlag is equal to 1 andsub_pic_cpb_params_in_pic_timiflg_sei_flag is equal to 0, decoding unitinformation SEI messages (present in BitstreamToDecode or availablethrough external means not specified) that are associated with decodingunits in the access unit and apply to TargetOp are selected as follows.

If bitstream-specific CPB operation is tested, the following applies:

If TargetDecLayerIdList contains all nuh_layer_id values present in thebitstream under test, non-nested buffering period, picture timing anddecoding unit information SEI messages are selected.

Otherwise, buffering period, picture timing and decoding unitinformation SEI messages included in the scalable nesting SEI messagewith bitstream_subset_flag equal to 1 and applicable to TargetOP areselected.

Otherwise, buffering period. picture timing and decoding unitinformation SE! messages included in the bitstream partition nesting SEImessage applicable to the bitstream partition under test are selected.

7. A value of SchedSelIdx is selected as follows.

If bitstream-specific CPB operation is tested, the selected SchedSelIdxshall be in the range of 0 to cpb_cnt_minus1[HighestTid], inclusive,where cpb_cnt_minus1[HighestTid] is found in thesub_layer_hrd_parameters(HighestTid) syntax structure as selected above.

Otherwise, a SchedSelCombIdx is selected for the bitstream under testand used for each bitstream partition under test. The selectedSchedSelCombIdx shall be in the range of 0 tonum_bsp_sched_combinations_minus1, inclusive. The selected SchedSelIdxshall be equal to bsp comb_sched_idx[SchedSelCombIdx][j] of thebitstream partition HRD message applicable to TargetOp where j is theindex of the bitstream partition under test.

When the coded picture in access unit 0 has nal_unit_type equal toCRA_NUT or BLA_W_LP, and irap_cpb_params_present_flag in the selectedbuffering period SEI message is equal to 1, either of the followingapplies for selection of the initial CPB removal delay and delay offset:

If NalHrdModeFlag is equal to 1, the default initial CPB removal delayand delay offset represented by nal_initial_cpbremoval_delay[SchedSelIdx] andnal_initial_cpb_removal_offset[SchedSelIdx], respectively, in theselected buffering period SEI message are selected. Otherwise, thedefault initial CPB removal delay and delay offset represented byvcl_initial_cpb removal_delay[SchedSelIdx] andvcl_initial_cpb_removal_offset[SchedSelIdx], respectively, in theselected buffering period SEI message are selected. The variableDefaultInitCpbParamsFlag is set equal to 1.

If NalHrdModeFlag is equal to 1, the alternative initial CPB removaldelay and delay offset represented bynal_initial_alt_cpb_removal_delay[SchedSelIdx] andnal_initial_alt_cpb_removal_offset[SchedSelIdx], respectively, in theselected buffering period SET message are selected. Otherwise, thealternative initial CPB removal delay and delay offset represented byvcl_initial_alt_cpb_removal_delay[SchedSelIdx] andvcl_initial_alt_cpb_removal_offset[SchedSelIdx], respectively, in theselected buffering period SEI message are selected. The variableDefaultInitCpbParamsFlag is set equal to 0, and the RASL access unitsassociated with access unit 0 are discarded from BitstreamToDecode andthe remaining bitstream is assigned to BitstreamToDecode.

9. For bitstream-partition-specific CPB operation. SubPicHrdFlag is setequal to 1. For bitstream-specific CPB operation. Whensub_pic_hrd_params_present_flag in the selected hrd_parameters( ) syntaxstructure is equal to 1, the CPB is scheduled to operate either at theaccess unit level (in which case the variable SubPicHrdFlag is set equalto 0) or at the sub-picture level (in which case the variableSubPicHrdFlag is set equal to 1).

For each operation point under test, the number of bitstream conformancetests for bitstream-specific CPB operation to be performed is equal ton0*n1*(n2*2+n3)*n4, where the values of n0, n1, n2, n3, and n4 arespecified as follows:

n0 is derived as follows:

-   -   If BitstreamToDecode is a Type I bitstream, n0 is equal to 1.    -   Otherwise (BitstreamToDecode is a Type II bitstream), n0 is        equal to 2.

n1 is equal to cpb_cnt_minus1[HighestTid]+1.

n2 is the number of access units in BitstreamToDecode that each isassociated with a buffering period SEI message applicable to TargetOpand for each of which both of the following conditions are true:

nal_unit_type is equal to CRA_NUT or BLA_W_LP for the VCL NAL units;

The associated buffering period SEI message applicable to TargetOp hasirap_cpb_params_present_flag equal to 1.

n3 is the number of access units in BitstreamToDecode that each isassociated with a buffering period SEI message applicable to TargetOpand for each of which one or both of the following conditions are true:

nal_unit_type is equal to neither CRA_NUT nor BLA_W_LP for the VCL NALunits;

The associated buffering period SEI message applicable to TargetOp hasirap_cpb_params_present_flag equal to 0.

If sub_pic_hrd_params_present_flag in the selected hrd_parameters( )syntax structure is equal to 0, n4 is equal to 1;

Otherwise, n4 is equal to 2.

When BitstreamToDecode is a Type II bitstream, the following applies:

If the sub_layer_hrd_parameters(HighestTid) syntax structure thatimmediately follows the condition “if (vcl_hrd_parameters_presentflag)”is selected, the test is conducted at the Type I conformance point shownin FIG. 7, and only VCL and filler data NAL units are counted for theinput bit rate and CPB storage.

Otherwise, (the sub_layer_hrd_parameters(HighestTid) syntax structurethat immediately follows the condition“if(nal_hrd_parameters_present_flag)” is selected, the test is conductedat the Type II conformance point shown in FIG. 7, and all bytes of theType II bitstream, which may be a NAL unit stream or a byte stream, arecounted for the input bit rate and CPB storage.

NAL HRD parameters established by a value of SchedSelIdx for the Type IIconformance point shown in FIG. 7 are sufficient to also establish VCLHRD conformance for the Type I conformance point shown in FIG. 7 for thesame values of InitCpbRemovalDelay[SchedSelIdx], BitRate[SchedSelIdx]and CpbSize[SchedSelIdx] for the VBR case (cbr_flag[SchedSelIdx] equalto 0). This is because the data flow into the Type I conformance pointis a subset of the data flow into the Type II conformance point andbecause, for the VBR case, the CPB is allowed to become empty and stayempty until the time a next picture is scheduled to begin to arrive. Forexample, when decoding a CVS conforming to one or more of the profilesusing a decoding process, when NAL HRD parameters are provided for theType II conformance point that not only fall within the bounds set forNAL HRD parameters for profile conformance but also fall within thebounds set for VCL HRD parameters for profile conformance, conformanceof the VCL HRD for the Type I conformance point is also assured to fallwithin specified bounds.

All VPSs, SPSs and PPSs referred to in the VCL NAL units, and thecorresponding buffering period, picture timing and decoding unitinformation SET messages shall be conveyed to the HRD, in a timelymanner, either in the bitstream (by non-VCL NAL units), or by othermeans.

The specification for “presence” of non-VCL NAL units that contain VPSs,SPSs, PPSs, buffering period SET messages, picture timing SET messages,or decoding unit information SEI messages is also satisfied when thoseNAL units (or just some of them) are conveyed to decoders (or to theHRD) by other means. For the purpose of counting bits, only theappropriate bits that are actually present in the bitstream are counted.

As an example, synchronization of such a non-VCL NAL unit, conveyed bymeans other than presence in the bitstream, with the NAL units that arepresent in the bitstream, are able to be achieved by indicating twopoints in the bitstream, between which the non-VCL NAL unit would havebeen present in the bitstream, had the encoder decided to convey it inthe bitstream.

When the content of such a non-VCL NAL unit is conveyed for theapplication by some means other than presence within the bitstream, therepresentation of the content of the non-VCL NAL unit is not required touse the same syntax as specified.

When HRD information is contained within the bitstream, it is possibleto verify the conformance of a bitstream to the requirements of thisclause based solely on information contained in the bitstream. When theHRD information is not present in the bitstream, as is the case for all“stand-alone” Type I bitstreams, conformance is verified when the HRDdata are supplied by some other means.

For bitstream-specific CPB operation, The HRD contains a coded picturebuffer (CPB), an instantaneous decoding process, a decoded picturebuffer (DPB), and output cropping as shown in FIG. 8.

For bitstream-partition-specific CPB operation. the HRD contains abitstream demuiltiplexer (optionally present), two or more bitstreampartition buffers (BPB), two or more instantaneous decoding processes, adecoded picture buffer (DPB), and output cropping as shown in FIG. 9.

For each bitstream conformance test, the CPB size (number of bits) forbitstream-specific CPB operation and tile BPB size forbitstream-partition-specific CPB operation is CpbSize[SchedSelIdx],where SchedSelIdx and the HRD parameters are specified above in thisclause. The DPB size (number of picture storage buffers) isspsmax_dec_pic_buffering_minus1[HighestTid]+1.

The variable SubPicHrdPreferredFlag is either specified by externalmeans, or when not specified by external means, set equal to 0.

When the value of the variable SubPicHrdFlag has not been set by step 9above in this clause, it is derived as follows:

SubPicHrdFlag=SubPicHrdPreferredFlag && sub_pic_hrd_params_present_flag

If SubPicHrdFlag is equal to 0, the HRD operates at access unit leveland each decoding unit is an access unit. Otherwise the HRD operates atsub-picture level and each decoding unit is a subset of an access unit.

If the HRD operates at access unit level, each time a decoding unit thatis an entire access unit is removed from the CPB. Otherwise (the HRDoperates at sub-picture level), each time a decoding unit that is asubset of an access unit is removed from the CPB. In both cases, eachtime an entire decoded picture is output from the DPB, though thepicture output time is derived based on the differently derived CPBremoval times and the differently signaled DPB output delays.

The following is specified for expressing the constraints in this annex:

Each access unit is referred to as access unit n, where the number nidentifies the particular access unit. Access unit 0 is selected perstep 4 above. The value of n is incremented by 1 for each subsequentaccess unit in decoding order.

Each decoding unit is referred to as decoding unit m, where the number midentifies the particular decoding unit. The first decoding unit indecoding order in access unit 0 is referred to as decoding unit 0. Thevalue of m is incremented by 1 for each subsequent decoding unit indecoding order.

The numbering of decoding units is relative to the first decoding unitin access unit 0.

Picture n refers to the coded picture or the decoded picture of accessunit n. The HRD operates as follows:

The HRD is initialized at decoding unit 0, with the both the CPB and theDPB and each BPB being set to be empty (the DPB fullness is set equal to0).

After initialization, the HRD is not initialized again by subsequentbuffering period SET messages.

For bitstream-specific CPB operation, data associated with decodingunits that flow into the CPB according to a specified arrival scheduleare delivered by the HSS. For bitstream-partition-specific CPBoperation, data associated with decoding units that flow into the BPBaccording to a specified arrival schedule are delivered by an HBPS.

When bitstream-partition-specific CPB operation is used, each bitstreampartition with index j is processed with SchedSelIdx equal tobsp_comb_schedidx[SchedSelCombIdx][j] of the bitstream partition HRDmessage applicable to TargetOp and the HSS replaced by the HPBS.

The data associated with each decoding unit is removed and decodedinstantaneously by the instantaneous decoding process at the CPB removaltime of the decoding unit.

Each decoded picture is placed in the DPB.

A decoded picture is removed from the DPB when it becomes no longerneeded for inter prediction reference and no longer needed for output.

HSS, HBPS and HRD information concerning the number of enumerateddelivery schedules and their associated bit rates and buffer sizes isspecified in clauses. The HRD is initialized as specified by thebuffering period SET message. The removal timing of decoding units fromthe CPB and output timing of decoded pictures from the DPB is specifiedusing information in picture timing SET messages or in decoding unitinformation SET messages. All timing information relating to a specificdecoding unit shall arrive prior to the CPB removal time of the decodingunit.

The requirements for bitstream conformance are specified, and the HRD isused to check conformance of bitstreams as specified above in thisclause and to check conformance of decoders as specified.

While conformance is guaranteed under the assumption that allpicture-rates and clocks used to generate the bitstream match exactlythe values signaled in the bitstream, in a real system each of these mayvary from the signaled or specified value.

All the arithmetic described herein is performed with real values, sothat no rounding errors propagate. For example, the number of bits in aCPB just prior to or after removal of a decoding unit is not necessarilyan integer.

The variable ClockTick is derived as follows and is called a clock tick:

ClockTick=vui_num_units_in_tick÷vui_time_scale

The variable ClockSubTick is derived as follows and is called a clocksub-tick:

ClockSubTick=ClockTick÷(tick_divisor_minus2+2)

Operation of Coded Picture Buffer (CPB) and Bitstream Partition Buffer(BPB)

The specifications in this clause apply independently to each set of CPBparameters that is present and to both the Type I and Type IIconformance points shown in FIG. 7, and the set of CPB parameters isselected as specified.

Timing of Decoding Unit Arrival

If SubPicHrdFlag is equal to 0, the variable subPicParamsFlag is setequal to 0, and the process specified in the remainder of this clause isinvoked with a decoding unit being considered as an access unit, forderivation of the initial and final CPB arrival times for access unit n.

Otherwise (SubPicHrdFlag is equal to 1), the process specified in theremainder of this clause is first invoked with the variablesubPicParamsFlag set equal to 0 and a decoding unit being considered asan access unit, for derivation of the initial and final CPB arrivaltimes for access unit n, and then invoked with subPicParamsFlag setequal to 1 and a decoding unit being considered as a subset of an accessunit, for derivation of the initial and final CPB arrival times for thedecoding units in access unit n.

The variables InitCpbRemovalDelay[SchedSelIdx] andInitCpbRemovalDelayOffset[SchedSelIdx] are derived as follows:

If one or more of the following conditions are true,InitCpbRemovalDelay[SchedSelIdx] andInitCpbRemovalDelayOffset[SchedSelIdx] are set equal to the values ofthe buffering period SET message syntax elementsnal_initial_alt_cpb_removal_delay[SchedSelIdx] andnal_initial_alt_cpb_removal_offset[SchedSelIdx], respectively, whenNalHrdModeFlag is equal to 1, orvcl_initial_alt_cpb_removal_delay[SchedSelIdx] andvcl_initial_alt_cpb_removal_offset[SchedSelIdx], respectively, whenNalHrdModeFlag is equal to 0, where the buffering period SET messagesyntax elements are selected as specified:

Access unit 0 is a BLA access unit for which each coded picture hasnal_unit_type equal to BLA_W_RADL or BLA_N_LP, and the value ofirap_cpb_params_present_flag of the buffering period SET message isequal to 1.

Access unit 0 is a BLA access unit for which the coded picture hasnal_unit_type equal to BLA_W_LP or is a CRA access unit, and the valueof irap_cpb_params_present_flag of the buffering period SEI message isequal to 1, and one or more of the following conditions are true:

UseAltCpbParamsFlag for access unit 0 is equal to 1.

DefaultInitCpbParamsFlag is equal to 0.

The value of subPicParamsFlag is equal to 1.

Otherwise, InitCpbRemovalDelay[SchedSelIdx] andInitCpbRemovalDelayOffset[SchedSelIdx] are set equal to the values ofthe buffering period SEI message syntax elementsnal_initial_cpb_removal_delay[SchedSelIdx] andnal_initial_cpb_removal_offset[SchedSelIdx], respectively, whenNalHrdModeFlag is equal to 1, orvcl_initial_cpb_removal_delay[SchedSelIdx] andvcl_initial_cpb_removal_offset[SchedSelIdx], respectively, whenNalHrdModeFlag is equal to 0, where the buffering period SEI messagesyntax elements are selected as specified.

Variables BspSyncFlag[bspIdx] [m], BspSyncBspIdx[bspIdx] [m] andBspSyncDuIdx[bspIdx] [m] are derived for bitstream-partition-specificCPB operation for bitstream partitions with index bspIdx greater than 0as follows:

If du_based_bpb_sync_flag[SchedSelCombIdx] of the bitstream partitionHRD message applicable to TargetOp is equal to 1, the following applies

If the DU preceding, in decoding order within tile bitstream, tile m-thDU of bitstream partition with index bspIdx belongs to to a bitstreampartition with all index prevBspIdx less than bspIdx,BspSyncFlag[bspIdx][m] is set equal to 1. BspSyncBspIdx[bspIdx][m] isset equal to 1, BspSyncDuIdx[bspIdx] [m] is set equal to tile decodingunit index of the preceding DU within bitstream partition with indexprevBspIdx.

Otherwise, BspSyncFlag[bspIdx][m] is set equal to 0.

Otherwise, du_based_bpb_sync_flag[SchedSelCombIdx] of the bitstreampartition HRD message applicable to TargetOP is equal to 0), thefollowing applies:

If the DU is the first DU, in decoding order, of a picture and anypicture in the inter-layer reference picture set has a nuh_layer_idvalue that belongs to a bitstream partition with a smaller index thanbspIdx, BspSyncFlag[bspIdx][m] is set equal to 1,BspSyncBspIdx[bspIdx][m] is the greatest bitstream partition indexrefBspIdx smaller than bspIdx associated with the nuh_layer_id valuerefLayerId of any picture in the inter-layer reference picture set, andBspSyncDuIdx[bspIdx] [m] is set equal to the decoding unit index (withinthe bitstream partition with index refBspIdx) of the last decoding unitof the picture with nuh_layer_id equal to refLayerId in the same accessunit as tile m-th DU of bitstream partition with index bspIdx.

Otherwise, BspSyncFlag[bspIdx][m] is set equal to 0.

The time at which the first bit of decoding unit m begins to enter theCPB is referred to as the initial arrival time initArrivalTime[m].

If bitstream-specific CPB operation is used, decoding units are indexedin decoding order within the bitstream. Otherwise(bitstream-partition-specific CPB operation is used), decoding units areindexed in decoding order with each bitstream partition, and thus adecoding unit may be identified with the pair of the bitstream partitionindex bspIdx and the decoding unit index m within the bitstreampartition.

The initial arrival time of decoding unit m is derived as follows:

If the decoding unit is decoding unit 0 (e.g., m=0) and eitherbitstream-specific CPB operation is used or the decoding unit belongs tothe base bitstream partition, initArrivalTime[0]=0.

Otherwise, if the decoding unit is decoding unit 0,bitstream-partition-specific CPB operation is used, the decoding unitdoes not belong to the base bitstream partition andcbr_flag[SchedSelIdx] is equal to 1, the following applies:

initArrivalTime [0] is obtained from the bitstream partition initialarrival tune SEI message applicable to TargetOp.

Otherwise, the following applies:

-   -   The variable bspSyncFlag is derived as follows:    -   If bitstream-specific CPB operation is used or the decoding unit        belongs to the base bitstream partition or BspSyncFlag[bspIdx]        [m] is equal to 0, bspSyncFlag is set equal to 0.    -   Otherwise, bspSyncFlag is set equal to 1.

If cbr_flag[SchedSelIdx] is equal to 1, the initial arrival time fordecoding unit m is equal to the final arrival time (which is derivedbelow) of decoding unit m−1, e.g.,

if( !subPicParamsFlag)  initArrivalTime[m] = AuFinalArrivalTime[m−1]else  initArrivalTime[m] = DuFinalArrivalTime[m−1]

When bspSyncFlag is equal to 1, it is a requirement of bitstreamconformance that initArrivalTime[m] 1 is greater than or equal to thefinal arrival time of the decoding unit with index BspSyncDuIdx[bspIdx][m] in the bitstream partition with index BspSyncBspIdx1[bspIdx][m].

Otherwise (cbr_flag[SchedSelIdx] is equal to 0), the initial arrivaltime for decoding unit m is derived as follows:

 if(!subPicParamsFlag)   initArrivalTime[m] = Max(AuFinalArrivalTime[m −1], initArrivalEarliestTime[m])   else   initArrivalTime[m] =Max(DuFinalArrivalTime[m − 1], initArrivalEarliestTime[m],refFinalArrivalTime)  where initArrivalEariiestTime[m]is derived asfollows:  The variable tmpNominalRemovalTime is derived as follows:  if(!subPicParamsFlag)    tmpNominalRemovalTime =AuNominalRemovalTime[m]   else    tmpNominalRemovalTime =DuNominalRemovalTime[m]where AuNominalRemovalTime[m] and DuNominalRemovalTime[m] are thenominal CPB removal time of access unit m and decoding unit m,respectively, as specified.

If decoding unit m is not the first decoding unit of a subsequentbuffering period, initArrivalEarliestTime[m] is derived as follows:

initArrivalEarliestTime[m]=tmpNominalRemovalTime−(InitCpbRemovalDelay[SchedSelIdx]+InitCpbRemovalDelayOffset[SchedSelIdx])+90000

Otherwise (decoding unit m is the first decoding unit of a subsequentbuffering period), initArrivalEarliestTime[m] is derived as follows:

initArrivalEarliestTime[m]=tmpNominalRemovalTime−(InitCpbRemovalDelay[SchedSelIdx]+90000)

and where refFinalArrivalTime is the final arrival time of the decodingunit with index BspSyncDuIdx[bspIdx] [m] in the bitstream partition withindex BspSyncBspIdx[bspIdx] [m].

The final arrival time for decoding unit m is derived as follows:

 if (!subPicParamsFlag)   AuFinalArrivalTime[m] = initArrivalTime[m] +sizeInbits[m] ÷ BitRate[SchedSelIdx]  else

DuFinalArrivaltime[m]=initArrivalTime[m]+sizeInbits[m]+BitRate[SchedSelIdx]where sizeInbits[m] is the size in bits of decoding unit m, counting thebits of the VCL NAL units and the filler data NAL units for the Type Iconformance point or all bits of the Type II bitstream for the Type IIconformance point, where the Type I and Type II conformance points areas shown in FIG. 7.

The values of SchedSelIdx, BitRate[SchedSelIdx], andCpbSize[SchedSelIdx] are constrained as follows:

If the content of the selected hrd_parameters( ) syntax structures forthe access unit containing decoding unit m and the previous access unitdiffer, the HSS selects a value SchedSelIdx1 of SchedSelIdx from amongthe values of SchedSelIdx provided in the selected hrd_parameters( )syntax structures for the access unit containing decoding unit m thatresults in a BitRate[SchedSelIdx1] or CpbSize[SchedSelIdx1] for theaccess unit containing decoding unit m. The value ofBitRate[SchedSelIdx1] or CpbSize[SchedSelIdx1] may differ from the valueof BitRate[SchedSelIdx0] or CpbSize[SchedSelIdx0] for the valueSchedSelIdx0 of SchedSelIdx that was in use for the previous accessunit.

Otherwise, the HSS continues to operate with the previous values ofSchedSelIdx, BitRate[SchedSelIdx] and CpbSize[SchedSelIdx].

When the HSS selects values of BitRate[SchedSelIdx] orCpbSize[SchedSelIdx] that differ from those of the previous access unit,the following applies:

The variable BitRate[SchedSelIdx] comes into effect at the initial CPBarrival time of the current access unit.

The variable CpbSize[SchedSelIdx] comes into effect as follows:

-   -   If the new value of CpbSize[SchedSelIdx] is greater than the old        CPB size, it comes into effect at the initial CPB arrival time        of the current access unit.    -   Otherwise, the new value of CpbSize[SchedSelIdx] comes into        effect at the CPB removal time of the current access unit.

Scalable Nesting SEI Message Semantics

The scalable nesting SEI message provides a mechanism to associate SEImessages with bitstream subsets corresponding to various operationpoints or with specific layers or sub-layers. A scalable nesting SEImessage contains one or more SEI messages. bitstream_subset_flag equalto 0 specifies that the SEI messages contained in the scalable nestingSEI message apply to specific layers or sub-layers.bitstream_subset_flag equal to 1 specifies that the SEI messagescontained in the scalable nesting SEI message apply to one or moresub-bitstreams resulting from a sub-bitstream extraction process withinputs based on the syntax elements of the scalable nesting SEI messageas specified below.

When a buffering period SEI message, a picture timing SEI message, or adecoding unit information SEI message is contained in the scalablenesting SEI message with bitstream_subset_flag equal to 0, they apply tothe bitstream partition including layers included in the listnestingLayerIdList[i] as derived herein.

Table 10 shows VPS VUI syntax.

vps_vui( ) { Descriptor  . . .  num_bitstream_partitions_minus1 ue(v) if(num_bitstream_partitions_minus1)   vps_vui_bsp_hrd( )  . . . }num_bitstream_partitions_minus1 equal to 0 specifies nobitstream-partition-specific HRD parameters are present.num_bitstream_partitions_minus1 plus 1 specifies the number of bitstreampartitions for which HRD parameters are specified in the bitstream.Table 11 shows another VPS VUI syntax.

vps_vui( ) { Descriptor  . . .  vps_vui_bsp_hrd_present_flag u(1) if(vps_vui_bsp_hrd_present_flag)   vps_vui_bsp_hrd( )  . . . }vps_vui_bsp_hrd_present_lag equal to U specifies that nobitstream-partition-specific HRD parameters are present.vps_vui_bsp_hrd_present_flag equal to 1 specifies thatbitstream-specific HRD parameters are present.Table 12 shows vps_vui_bsp_hrd syntax.

vps_vui_bsp_hrd( ) { Descriptor  vps_num_bsp_hrd_parameters_minus1 ue(v) for(i=0; i<=vps_num_bsp_hrd_parameters_minus1; i++) {   if(i > 0)   bsp_cprms_present_flag[i] u(1)  hrd_parameters(bsp_cprms_present_flag[i], vps_max_sub_layers_minus1) }  for(h=1; h<=vps_num_layer_sets_minus1; h++) {  num_bitstream_partitions[h] ue(v)   for(i=0;i<=num_bitstream_partitions[h]; i++) {    for(j=0;j<=vps_max_layers_minus1; j++)     if(layer_id_included_flag[h][j]     layer_in_bsp_flag[h][i][j] u(1)    }   }  if(num_bitstream_partitions{    num_bsp_sched_combinations[h] ue(v)   for(i=0; i<=num_bsp_sched_combinations[h]; i++) {    du_based_bpb_sync_flag[h][i] u(1)     for(j=0;j<=num_bitstream_partitions[h]; j++) {      bsp_comb_hrd_idx[h][i][j]ue(v)      bsp_comb_sched_idx[h][i][j] ue(v)     }    }   }  } }vps_num_bsp_hrd_parameters_minus1 plus 1 specifies the number ofhrd_parameters( ) syntax structures present within the vps_vui_bsp_hrd() syntax structure.bsp_cprms_present_flag[i] equal to 1 specifies that the HRD parametersthat are common for all sub-layers are present in the i-thhrd_parameters( ) syntax structure in the vps_vui_bsp_hrd( ) syntaxstructure. bsp_cprms_present_flag[i] equal to 0 specifies that the HRDparameters that are common for all sub-layers are not present in thei-th hrd_parameters( ) syntax structure in the vps_vui_bsp_hrd( ) syntaxstructure and are derived to be the same as the (i−1)-thhrd_parameters( ) syntax structure in the in the vps_vui_bsp_hrd( )syntax structure.bsp_cprms_present_flag[0] is inferred to be equal to 1.num_bitstream_partitions[h] specifies the number of bitstream partitionsfor which HRD parameters are specified for the layer set with index h.layer_in_bsp_flag[h][i][j] specifies that the layer with index j is apart of bitstream partition with index i within a layer set with indexh. The bitstream partition with index j shall not include direct orindirect reference layers of any layers in bitstream partition i for anyvalues of i and j in the range of 0 to num_bitstream_partitions[h]−1,inclusive, such that i is less than j.num_bsp_sched_combinations[h] specifies the number of combinations ofdelivery schedules and hrd_parameters( ) specified for bitstreampartitions for layer set with index h.du_based_bpb_sync_flag[h][i] affects the CPB operation when the i-thcombination of delivery schedules and hrd_parameters( ) is used.bsp_comb_hrd_idx[h][i][j] specifies the index of hrd_parameters( )within the vps_vui_bsp_hrd( ) syntax structure used in the i-thcombination.bsp_comb_sched_idx[h][i][j] specifies the index of delivery schedulewithin the hrd_parameters( ) syntax structure with the indexbsp_comb_hrd_idx[h][i][j] that is used in the i-th combination.Table 13 shows another vps_vui_bsp_hrd syntax.

vps_vui_bsp_hrd( ) { Descriptor  vps_num_bsp_hrd_parameters_minus1 ue(v) for(i=0; i<=vps_num_bsp_hrd_parameters_minus1; i++) {   if(i > 0)   bsp_cprms_present_flag[i] u(1)  hrd_parameters(bsp_cprms_present_flag[i], vps_max_sub_layers_minus1) }  for(h=0; h<=vps_max_layers_sets_minus1; h++) {   for(i=0;i<=num_bitstream_partitions_minus1; i++) {    for(j=0;j<=vps_max_layers_minus1; j++)     bsp_idx_to_bsp_layer_idx[h][i][j]u(v)    }   }  }  for(h=1; h<=vps_num_layer_sets_minus1; h++) {  num_bsp_sched_combinations[h] ue(v)   for(i=0;i<=num_bsp_sched_combinations[h]; i++) {    du_based_bpb_sync_flag[h][i]u(1)    for(j=0; j<=num_bitstream_partitions_minus1; j++) {    bsp_comb_hrd_idx[h][i][j] ue(v)     bsp_comb_sched_idx[h][i][j]ue(v)    }   }  } }vps_num_bsp_hrd_parameters_minus1 plus 1 specifies the number ofhrd_parameters( ) syntax structures present within the vps_vui_bsp_hrd() syntax structure.bsp_cprms_present_flag[i] equal to 1 specifies that the HRD parametersthat are common for all sub-layers are present in the i-thhrd_parameters( ) syntax structure in the vps_vui_bsp_hrd( ) syntaxstructure. bsp_cprms_present_flag[i] equal to 0 specifies that the HRDparameters that are common for all sub-layers are not present in thei-th hrd_parameters( ) syntax structure in the vps_vui_bsp_hrd( ) syntaxstructure and are derived to be the same as the (i−1)-th hrd_parameters() syntax structure in the in the vps_vui_bsp_hrd( ) syntax structure.bsp_cprms_present_flag[0] is inferred to be equal to 1.bsp_idx_tobsp_layer_idx[h] [i] [j] specifies that the layer with index jis a part of bitstream partition with index i within a layer set withindex h.num_bsp_sched_combinations [h] specifies the number of combinations ofdelivery schedules and hrd_parameters( ) specified for bitstreampartitions for layer set with index h.du_based_bpb_sync_flag[h][i] affects the CPB operation when the i-thcombination of delivery schedules and hrd_parameters( ) is used.bsp_comb_hrd_idx[h] [i] [j] specifies the index of hrd_parameters( )within the vps_vui_bsp_hrd( ) syntax structure used in the i-thcombination.bsp_comb_sched_idx[h][i][j] specifies the index of delivery schedulewithin the hrd_parameters( ) syntax structure with the indexbsp_comb_hrd_idx[h][i][j] that is used in the i-th combination.Table 14 shows bitstream partition HRD SEI message syntax.

bsp_hrd(payloadSize) { Descriptor  num_bitstream_partitions_minus1 ue(v) for(i=0; i<=num_bitstream_partitions_minus1; i++) {   for(j=0;j<=vps_max_layers_minus1 j++)   if(layer_id_included_flag[nesting_op_idx[0]][j]) {    layer_in_bsp_flag[i][j] u(1)   }  }  num_bsp_hrd_paramters_minus1ue(v)  for(i=0; i<=num_bsp_hrd_parameters_minus1; i++) {   if (i>0)   bsp_cprms_present_flag[i] u(1)  hrd_parameters(bsp_cprms_present_flag[i],nesting_max_temporal_id_plus[0] − 1)  } num_bsp_sched_combinations_minus1 ue(v)  for(i=0;i<==numb_bsp_sched_combinations_minus1; i++) {  du_based_bpb_sync_flag[i] u(1)   for(j=0;j<==num_bitstream_partitions_minus1; j++) {    bsp_comb_hrd_idx[i][j]ue(v)    bsp_comb_sched_idx[i][j] ue(v)    }   } }

Bitstream Partition HRD SEI Message Semantics

The bitstream partition HRD SEI message specifies HRD parameters forbitstream-partition-specific CPB operation.

When present, this SEI message shall be contained within a scalablenesting SEI message in an initial IRAP access unit. When this SEImessage is contained in a scalable nesting SEI message, it shall be theonly nested SEI message. In the scalable nesting SEI message containingthis SEI message, bitstream_subset_flag shall be equal to 1,nesting_op_flag is equal to 1, default_op_flag shall be equal to 0 andnesting num_ops_minus1 shall be equal to 0.

The message applies to the layer set specified by the listnestingLayerIDList[0]. num_bitstream_partitions_minus1 plus 1 specifiestile number of bitstream partitions for which HRD parameters arespecified.

layer_in_bsp_flag[i][j] specifies that the layer with index j is a partof the bitstream partition with index i.

For bitstream conformance, the bitstream partition with index j shallnot include direct or indirect reference layers of any layers inbitstream partition i for any values of i and j in the range of 0 tonum_bitstream_partitions_minus1, inclusive, such that i is less than j.

num_bsp_hrd_parameters_minus1 plus 1 specifies the number ofhrd_parameters( ) syntax structures present within the SEI message.bsp_cprms_present_flag[i] equal to 1 specifies that the HRD parametersthat are common for all sub-layers are present in the i-th hrdparameters( ) syntax structure in the SEI message.bsp_cprms_present_flag[i] equal to 0 specifies that the HRD parametersthat are common for all sub-layers are not present in the i-th hrdparameters( ) syntax structure in the SEI message and are derived to bethe same as the (i−1)-th hrd_parameters( ) syntax structure in the SEImessage.bsp_cprms_present_flag[0] is inferred to be equal to 1.num_bsp_sched_combinations_minsu1 specifies the number of combinationsof delivery schedules and hrd_parameters( ) specified for bitstreampartitions.du_based_bpb_sync_flag[i] affects the CPB operation when the i-thcombination of delivery schedules and hrd_parameters( ) is used.bsp_comb_hrd_idx[i][j] specifies the index of hrd_parameters( ) withinthe SEI message used in the i-th combination.bsp_comb_sched_idx[i][j] specifies the index of delivery schedule withinthe hrd_parameters( ) syntax structure with the indexbsp_comb_hrd_idx[i][j] that is used in the i-th combination.Table 15 shows bitstream partition nesting SEI message syntax.

bsp_nesting(payloadSize) { Descriptor  bsp_idx ue(v) while(!byte_aligned( ))   bsp_nesting_zero_bit /*equal to 0*/ u(1)  do  sei_message( )  while(more_rbsp_data( )) }Tile bitstream partition nesting SEI message provides a mechanism toassociate SEI message with a bitstream partition of a layer set.When present this SEI message shall be contained within a scalablenesting SEI message. When this SEI message is contained in a scalablenesting SEI message, it shall be the only nested SEI message. In thescalable nesting SEI message containing this SEI messagebitstream_subset_flag shall be equal to 1, nesting_op_flag is equal to1, default_op_flag shall be equal to 0 and nesting_num_ops_minus1 shallbe equal to 0.The associated bitstream partition HRD message for the bitstreampartition nesting SEI message is the preceding bitstream partition HRDmessage, in decoding order, that is nested in a scalable nesting SEImessage with the same value of nesting_op_idx[0] as that for thescalable nesting SEI message containing the bitstream partition nestingSEI message. It is a requirement of bitstream conformance that whenbitstream partition nesting SEI message is present, it shall have anassociated bitstream partition HRD message within the same coded videosequence.A bitstream partition nesting SEI message contains one or more SEImessages. bsp_idx specifies the bitstream partition index among thebitstream partitions specified in the associated bitstream partition HRDmessage. This SEI message applies to the bitstream partition with indexbsp_idx.Table 16 shows bitstream partition initial arrival time SET messagesyntax.

bsp_initial_arrival_time(payloadSize) { Descriptor if(NalHrdBpPresentFlag)   for(i=0; i<SchedCombCnt; i++)   nal_initial_arrival_delay[i] u(v)  else   for(i=0; i<SchedCombCnt;i++)    vcl_initial_arrival_delay[i] u(v) }The bitstream partition initial arrival time SEI message specifies theinitial arrival delays to be used in the bitstream-partition-specificCPB operation.When present, the SEI message shall be contained within bitstreampartition nesting SEI message that is contained in a scalable nestingSEI message. The same bitstream partition SEI message shall also containa picture buffering SEI message.nal_initial_arrival_delay[i] specifies the initial BPB arrival delaysfor the i-th schedule combination of the bitstream partition to whichthis SEI message applies, when NAL HRD parameters are in use.vcl_initial_arrival_delay[i] specifies the initial BPB arrival delaysfor the i-th schedule combination of the bitstream partition to whichthe SEI message applies, when VCL HRD parameters are in use.

Demultiplexing Process for Deriving a Bitstream Partition

Inputs to the process are a bitstream, a layer identifier listbspLayerId[idx], and the number of layer identifiers numBspLayerId inthe layer index list bspLayerId[idx].Output of the process is a bitstream partition.Let variable minBspLayerId be the smallest value of bspLayerId[idx] withany value of idx in the range of 0 to numBspLayerId−1, inclusive.The output bitstream partition includes selected NAL units of the inputbitstream in the same order as they appear in the input bitstream. Thefollowing NAL units of the input bitstream are omitted from the outputbitstream partition, while the remaining NAL units of the inputbitstream are included in the output bitstream partition:

Omit all NAL units that have a nuh_layer_id_value other thanbspLayerId[idx] with any value of idx in the range of 0 tonumBspLayerId−1, inclusive.

Omit all SEI NAL units containing a scalable nesting SEI message forwhich no derived nesting LayerIdList[i] contains any layer identifiervalue equal to bspLayerId[idx] with any value of idx in the range of 0to numBspLayerId−1, inclusive.

Omit all SEI NAL units containing a scalable nesting SEI message forwhich a derived nestingLayerIdList[i] contains a layer identifier valueless than minBspLayerId.

FIG. 10 illustrates a flowchart of implementing buffer managementmethods according to some embodiments. In the step 1000, a video isacquired. The video is able to be acquired in any manner such as beingcaptured using a digital camcorder, downloaded from another device,generated by a device or any other implementation. In the step 1002, abitstream of the video is transmitted including signaling a maximum bitrate, average bit rate and maximum coded picture buffer size allocatedfor each layer in the bitstream. In the step 1004, an extension andparameters that are configured to be defined as whole-sequence level orsub-stream level are implemented. In the step 1006, a system buffer ismanaged for each layer independently without having to re-assemble lowerlayers. In the step 1008, buffers are managed for base and enhancementlayers of the video before re-assembly. In the step 1010, a multilayerdecoder is implemented. In the step 1012, buffering period and picturetiming supplemental enhancement information messages are associated witheach layer of the video. In the step 1014, a same picture timingsupplemental enhancement information message for all layers of the videofor synchronization of removal times within a layer set is utilized, anddifferent buffering period supplemental enhancement information messagesare utilized for each layer of the video. In some embodiments, fewer oradditional steps are implemented.

FIG. 11 illustrates a block diagram of an exemplary computing deviceconfigured to implement the buffer management methods according to someembodiments. The computing device 1100 is able to be used to acquire,store, compute, process, communicate and/or display information such asimages and videos. In general, a hardware structure suitable forimplementing the computing device 1100 includes a network interface1102, a memory 1104, a processor 1106, I/O device(s) 1108, a bus 1110and a storage device 1112. The choice of processor is not critical aslong as a suitable processor with sufficient speed is chosen. The memory1104 is able to be any conventional computer memory known in the art.The storage device 1112 is able to include a hard drive, CDROM, CDRW,DVD, DVDRW, High Definition disc/drive, ultra-HD drive, flash memorycard or any other storage device. The computing device 1100 is able toinclude one or more network interfaces 1102. An example of a networkinterface includes a network card connected to an Ethernet or other typeof LAN. The I/O device(s) 1108 are able to include one or more of thefollowing: keyboard, mouse, monitor, screen, printer, modem,touchscreen, button interface and other devices. Buffer managementapplication(s) 1130 used to perform the buffer management methods arelikely to be stored in the storage device 1112 and memory 1104 andprocessed as applications are typically processed. More or fewercomponents shown in FIG. 11 are able to be included in the computingdevice 1100. In some embodiments, buffer management hardware 1120 isincluded. Although the computing device 1100 in FIG. 11 includesapplications 1130 and hardware 1120 for the buffer management methods,the buffer management methods are able to be implemented on a computingdevice in hardware, firmware, software or any combination thereof. Forexample, in some embodiments, the buffer management applications 1130are programmed in a memory and executed using a processor. In anotherexample, in some embodiments, the buffer management hardware 1120 isprogrammed hardware logic including gates specifically designed toimplement the buffer management methods.

In some embodiments, the buffer management application(s) 1130 includeseveral applications and/or modules. In some embodiments, modulesinclude one or more sub-modules as well. In some embodiments, fewer oradditional modules are able to be included.

Examples of suitable computing devices include a personal computer, alaptop computer, a computer workstation, a server, a mainframe computer,a handheld computer, a personal digital assistant, a cellular/mobiletelephone, a smart appliance, a gaming console, a digital camera, adigital camcorder, a camera phone, a smart phone, a portable musicplayer, a tablet computer, a mobile device, a video player, a video discwriter/player (e.g., DVD writer/player, high definition discwriter/player, ultra high definition disc writer/player), a television,a home entertainment system, smart jewelry (e.g., smart watch) or anyother suitable computing device.

To utilize the buffer management methods, a device such as a digitalvideo camera is used to acquire a video. The buffer management methodsare automatically implemented during or after acquiring a video. Thebuffer management methods are able to be implemented automaticallywithout user involvement.

In operation, the buffer management methods are implemented to improveefficiency of processing videos. The methods simplify the complexity ofSTD buffer management for HEVC and make is easy to implement HEVC indeployed AVC/MPEG-2 networks (as legacy re-multiplexers are able to beused for re-purposing HEVC content). The buffers for base andenhancement layers are also able to be managed independently beforere-assembly and this simplifies the STD model. Re-assembly is stillimplemented before decoding an enhanced HEVC video stream.

Some Embodiments of Layer Based HRD Buffer Management for Scalable HEVC

-   1. A method programmed in a non-transitory memory of a device    comprising:    -   a. acquiring a video; and    -   b. transmitting a bitstream of the video including signaling a        maximum bit rate, average bit rate and maximum coded picture        buffer size allocated for each layer in the bitstream.-   2. The method of clause 1 further comprising implementing an    extension and parameters that are configured to be defined as    whole-sequence level or sub-stream level.-   3. The method of clause 2 further comprising managing a system    buffer for each layer independently without having to re-assemble    lower layers.-   4. The method of clause 1 further comprising managing buffers for    base and enhancement layers of the video before re-assembly.-   5. The method of clause 1 further comprising implementing a    multilayer decoder.-   6. The method of clause 1 further comprising associating buffering    period and picture timing supplemental enhancement information    messages with each layer of the video.-   7. The method of clause 1 further comprising utilizing a same    picture timing supplemental enhancement information message for all    layers of the video for synchronization of removal times within a    layer set, and utilizing different buffering period supplemental    enhancement information messages for each layer of the video.-   8. The method of clause 1 wherein the device is selected from the    group consisting of a personal computer, a laptop computer, a    computer workstation, a server, a mainframe computer, a handheld    computer, a personal digital assistant, a cellular/mobile telephone,    a smart phone, a smart appliance, a gaming console, a digital    camera, a digital camcorder, a camera phone, an portable music    player, a tablet computer, a video player, a DVD writer/player, a    high definition video writer/player, a television and a home    entertainment system.-   9. A system programmed in a non-transitory memory of a camera device    comprising:    -   a. a sensor configured for acquiring a video; and    -   b. a transmitting component configured for transmitting a        bitstream of the video including signaling a maximum bit rate,        average bit rate and maximum coded picture buffer size allocated        for each layer in the bitstream.-   10. The system of clause 9 wherein the transmitting component is    further configured for implementing an extension and parameters that    are configured to be defined as whole-sequence level or sub-stream    level.-   11. The system of clause 10 wherein the transmitting component is    further configured for managing a system buffer for each layer    independently without having to re-assemble lower layers.-   12. The system of clause 9 wherein the transmitting component is    further configured for managing buffers for base and enhancement    layers of the video before re-assembly.-   13. The system of clause 9 wherein the transmitting component is    further configured for implementing a multilayer decoder.-   14. The system of clause 9 wherein the transmitting component is    further configured for associating buffering period and picture    timing supplemental enhancement information messages with each layer    of the video.-   15. The system of clause 9 wherein the transmitting component is    further configured for utilizing a same picture timing supplemental    enhancement information message for all layers of the video for    synchronization of removal times within a layer set, and utilizing    different buffering period supplemental enhancement information    messages for each layer of the video.-   16. An apparatus comprising:    -   a. a sensor configured for acquiring a video;    -   b. a non-transitory memory for storing an application, the        application for: transmitting a bitstream of the video including        signaling a maximum bit rate, average bit rate and maximum coded        picture buffer size allocated for each layer in the bitstream;        and    -   c. a processing component coupled to the memory, the processing        component configured for processing the application.-   17. The apparatus of clause 16 wherein the application is further    configured for implementing an extension and parameters that are    configured to be defined as whole-sequence level or sub-stream    level.-   18. The apparatus of clause 17 wherein the application is further    configured for managing a system buffer for each layer independently    without having to re-assemble lower layers.-   19. The apparatus of clause 16 wherein the application is further    configured for managing buffers for base and enhancement layers of    the video before re-assembly.-   20. The apparatus of clause 16 wherein the application is further    configured for implementing a multilayer decoder.-   21. The apparatus of clause 16 wherein the application is further    configured for associating buffering period and picture timing    supplemental enhancement information messages with each layer of the    video.-   22. The apparatus of clause 16 wherein the application is further    configured for utilizing a same picture timing supplemental    enhancement information message for all layers of the video for    synchronization of removal times within a layer set, and utilizing    different buffering period supplemental enhancement information    messages for each layer of the video.

The present invention has been described in terms of specificembodiments incorporating details to facilitate the understanding ofprinciples of construction and operation of the invention. Suchreference herein to specific embodiments and details thereof is notintended to limit the scope of the claims appended hereto. It will bereadily apparent to one skilled in the art that other variousmodifications may be made in the embodiment chosen for illustrationwithout departing from the spirit and scope of the invention as definedby the claims.

1-22. (canceled)
 23. An encoding method, comprising: setting a supplemental enhancement information message for each of a base layer bitstream included in a bitstream and a combined base and enhancement layer bitstream included in the bitstream, for synchronization of removal times such that a number of bits that is present in the bitstream is counted to decode a particular set of bits of the bitstream; and generating the bitstream including the supplemental enhancement information message, wherein the base layer bitstream and the combined base and enhancement layer bitstream include hypothetical reference decoder parameters for each layer, the hypothetical reference decoder parameters comprise at least one of a bit rate or a Coded Picture Buffer (CPB) size, and a first buffer for the base layer bitstream and a second buffer for the combined base and enhancement layer bitstream are managed independently based on respective hypothetical reference decoder parameters.
 24. The encoding method according to claim 23, wherein the base layer bitstream utilizes a different profile than the combined base and enhancement layer bitstream.
 25. The encoding method according to claim 23, wherein the bitstream indicates the at least one of the bit rate or the CPB size allocated for each of the base layer bitstream and the combined base and enhancement layer bitstream.
 26. An encoding system, comprising: at least one processor configured to: set a supplemental enhancement information message for each of a base layer bitstream included in a bitstream and a combined base and enhancement layer bitstream included in the bitstream, for synchronization of removal times such that a number of bits that is present in the bitstream is counted to decode a particular set of bits of the bitstream; and generate the bitstream including the supplemental enhancement information message, wherein the base layer bitstream and the combined base and enhancement layer bitstream include hypothetical reference decoder parameters for each layer, the hypothetical reference decoder parameters comprise at least one of a bit rate or a Coded Picture Buffer (CPB) size, and a first buffer for the base layer bitstream and a second buffer for the combined base and enhancement layer bitstream are managed independently based on respective hypothetical reference decoder parameters.
 27. The encoding system according to claim 26, wherein the base layer bitstream utilizes a different profile than the combined base and enhancement layer bitstream.
 28. The encoding system according to claim 26, wherein the bitstream indicates the at least one of the bit rate or the CPB size allocated for each of the base layer bitstream and the combined base and enhancement layer bitstream. 